Dynamic detection of cross-document associations

ABSTRACT

Systems and methods are configured to generate a set of related document objects for a predictive entity and/or to generate an optimal document sequence for a set of related document objects. In one embodiment, a set of related document objects for a predictive entity is generated by processing entity metadata features associated with the predictive entity using an entity-document correlation machine learning model, and an optimal document sequence is generated for the set of related document objects by processing the set of related document objects using a document sequence optimization machine learning model.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. Non-Provisional patentapplication Ser. No. 17/015,511, filed Sep. 9, 2020, which isincorporated by reference herein in its entirety.

BACKGROUND

Embodiments of the present invention generally relate to improvingdocument retrieval efficiency and/or document retrieval reliability indocument management systems.

BRIEF SUMMARY

In general, embodiments of the present disclosure provide methods,apparatus, systems, computing devices, computing entities, and/or thelike for generating a subset of related document objects for apredictive entity. In various embodiments, a set of entity metadatafeatures associated with the predictive entity instance of thepredictive entity is identified and processed using an entity-documentcorrelation machine learning model to identify a subset of relateddocument objects for the predictive entity instance from a set ofreference document objects.

In particular embodiments, the entity-document correlation machinelearning model is configured at least in part based on a set of inferredannotative relationships between a set of input index document objectsand the set of reference document objects. In some embodiments, eachinferred annotative relationship is associated with a correspondinginput index document object and describes a related reference documentsubset of the set of reference document objects for the correspondinginput index document object. In addition, the set of entity metadatafeatures associated with the predictive entity instance of thepredictive entity may include at least one of one or more demographicfeatures, one or more diagnostic features, and one or morepolicy-related features. Accordingly, one or more actions may beperformed based on the subset of related document objects identified bythe model.

In various embodiments, the entity-document correlation machine learningmodel is developed based on the related reference document subset and aset of input index document metadata features determined for each inputindex document object of the set of input index document objects. Insome embodiments, the related reference document subset for an inputindex document object may be identified by processing the input indexdocument object using a cross-document annotation machine learning modelbased on a set of prior annotative relationships between a set of priorindex document objects and a set of prior reference document objects.

In addition, in some embodiments, the entity-document correlationmachine learning model is configured to generate a related documentsequence for the subset of related document objects that defines arelated document ordering value for each related document object in thesubset of related document objects. Accordingly, in these embodiments,each inferred annotative relationship may define a related referencedocument sequence for the related reference document subset that isassociated with the inferred annotative relationship. This relatedreference document sequence defines a reference document ordering valuefor each related reference document in the related reference documentsubset and can be used to train the entity-document correlation machinelearning model to generate the related document sequence.

Further, in some embodiments, each inferred annotative relationship maydescribe one or more cross-document annotations. In some embodiments,each cross-document annotation may be associated with a correspondingtext segment of one or more text segment associated with thecorresponding input index document object, and the one or more textsegments may be associated with a text segment order. In particularembodiments, the related reference document sequence defined by theinferred annotative relationship may be determined based on the textsegment order.

Furthermore, embodiments of the present disclosure provide methods,apparatus, systems, computing devices, computing entities, and/or thelike for generating an optimal document sequence for a set of inputdocument objects. In various embodiments, a set of sequence-relatedfeatures is determined for each input document object in the set ofinput document objects and processed using a document sequenceoptimization machine learning model to generate the optimal documentsequence. For instance, depending on the embodiment, the set ofsequence-related features for an input document object may include atleast one of a document identifier of the input document object, adocument creation date identifier of the input document object, adocument latest revision date identifier of the input document object,and the like.

In particular embodiment, the document sequence optimization machinelearning model is configured to generate a set of candidate documentsequences for the set of input document objects, generate a combinedsequence probability value for each candidate document sequence of theset of candidate document sequences based on a plurality of pair-wisedocument sequencing probabilities associated with the set of inputdocument objects, and select the optimal document sequence from the setof candidate document sequences based on the combined sequenceprobability values. Accordingly, one or more actions may be performedbased on the optimal document sequence generated by the model.

In some embodiments, each pair-wise document sequencing probability isassociated with an ordered document pair of a plurality of ordereddocument pairs of the set of input document objects. In someembodiments, each ordered document pair of the plurality of ordereddocument pairs may be associated with one or more prior document objectsof the set of input document objects and a posterior document object ofthe set of input document objects. Accordingly, each pair-wise documentsequencing probability describes an estimated probability that the oneor more prior document objects in the ordered document pair for thepair-wise document sequencing probability is immediately followed by theposterior document object in the ordered document pair for the pair-wisedocument sequencing probability.

In addition, in some embodiments, the set of related document objectsare associated with a prior document sequence and the document sequenceoptimization machine learning model is configured to generate thecombined sequence probability value for a corresponding candidatedocument sequence that corresponds to the prior document sequence bygenerating a raw combined sequence probability value for thecorresponding candidate document sequence and applying a prior selectionbonus to the raw combined sequence probability value. Finally, in someembodiments, the plurality of pair-wise document sequencingprobabilities may be determined based on historical user activity dataassociated with a corpus of monitored document objects.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

Having thus described the invention in general terms, reference will nowbe made to the accompanying drawings, which are not necessarily drawn toscale, and wherein:

FIG. 1 is an overview of aggregating information from differentdocuments according to various embodiments of the present disclosure;

FIG. 2 is a diagram of a system architecture in accordance with variousembodiments of the present disclosure;

FIG. 3 is a schematic of a computing entity that can be used inaccordance with various embodiments of the present disclosure;

FIG. 4 is an overview of a process for identifying a sequence ofdocuments in accordance with various embodiments of the presentdisclosure;

FIG. 5 is an example of an annotated document that can be used inaccordance with various embodiments of the present disclosure;

FIG. 6 is a process flow for aggregating information in accordance withvarious embodiments of the present disclosure;

FIG. 7 is a process flow for predicting a subset of related documentobjects in accordance with various embodiments of the presentdisclosure;

FIG. 8 is a process flow for predicting an optimal document sequence inaccordance with various embodiments of the present disclosure;

FIG. 9 is a process flow for parsing information downloaded from adocument in accordance with various embodiments of the presentdisclosure;

FIG. 10 is a process flow for extracting information needed for adocument in accordance with various embodiments of the presentdisclosure;

FIG. 11 is an example of a knowledge map that may be used in accordancewith various embodiments of the present disclosure.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

Various embodiments of the present invention now will be described morefully hereinafter with reference to the accompanying drawings, in whichsome, but not all embodiments of the inventions are shown. Indeed, theseinventions may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. The term “or” (also designated as “/”) is usedherein in both the alternative and conjunctive sense, unless otherwiseindicated. The terms “illustrative” and “exemplary” are used to beexamples with no indication of quality level. Like numbers refer to likeelements throughout.

Definitions of Certain Terms

The term “document object” may refer to a data object describing acollection of data items, such as a collection of text data items, acollection of image data items, and/or the like. Examples of documentobjects include Portable Document Format (PDF) files, Hyper-Text MarkupLanguage (HTML) source files, Microsoft Word documents, image files,and/or the like. In some embodiments, document objects describepre-authorization review guidelines, regulatory guidelines, knowledgelibraries, standard operating procedure (SOP) guidelines, and/or thelike.

The term “predictive entity” may refer to a data object associated witha set of related document objects, where detecting the correlation ofthe predictive entity with the set of related objects may be performedusing an entity-document correlation machine learning model. Forinstance, an example that is used throughout the disclosure isinvestigating information from various information sources for use by ahealth insurance provider in adjudicating an incoming case for a memberwho is seeking pre-authorization for a particular set of one or moremedical procedures and/or treatments. In some embodiments, thepredictive entity may be a member identifier with respect to which thepre-authorization task may be performed. In the noted example, thepredictive entity may provide initial information on the member as wellas information on the occurrence of a medical procedure and/or medicaltreatment in relation to the member.

The term “entity metadata features” may refer to a data object thatdescribes one or more features associated with a predictive entity thatmay be used in accordance with an entity-document correlation machinelearning model to detect a set of related document objects for thepredictive entity. For example, if the predictive entity is a memberidentifier, examples of entity metadata features include memberdiagnostic features, member policy-related features, and memberdemographic features. Examples of member diagnostic features includediagnosis codes associated with a member and Current ProceduralTerminology (CPT) codes associated with a member. Examples of memberpolicy-related features include set numbers, policy numbers, groupnumbers, and/or the like. Examples of member demographic featuresinclude member names and member identifier numbers.

The term “entity-document correlation machine learning model” may berefer to a data object that describes parameters and/or hyper-parametersof a machine learning model that is configured to identify a subset ofrelated document objects for a predictive entity based on entitymetadata features for the predictive entity. In some embodiments, inparticular embodiments, the entity-document correlation machine learningmodel is generated at least in part using a set of inferred annotativerelationships between a set of input index documents and a set ofinferred reference documents. An example of an entity-documentcorrelation machine learning model is a probabilistic deep learningmodel.

The term “index document object” may refer to a data object thatdescribes a document object having a portion that is deemed to refer toa particular reference document object. For example, an index documentobject may represent a case note of summarized information referred toby a reviewer during a past case when a member was seekingpre-authorization for a particular medical procedure and/or treatmentand a reference document object may represent an information sourceidentified in the case note used by the reviewer in adjudicating thecase. For example, if a text portion T1 of Document object D1 may bedeemed to refer to Document object D2, document object D1 is deemed tobe an index document object.

The term “reference document object” may refer a data object thatdescribes a document object that is referred to by a portion of an indexdocument object. Examples of reference document objects includepre-authorization review guidelines, regulatory guidelines, knowledgelibraries, standard operating procedure (SOP) guidelines, and/or thelike. For example, if a text portion T1 of Document object D1 is deemedto refer to Document object D2, document object D2 may be deemed to be areference document object.

The term “inferred annotative relationship” may refer to a data objectthat describes an inferred relationship between a portion of an indexdocument object and one or more related reference documents, where theinferred relationship may be detected using a cross-document annotationmachine learning model. For example, a particular inferred annotativerelationship may describe that a text portion T1 of Document object D1is deemed to refer to Document object D2. As another example, aparticular inferred annotative relationship may describe that a textportion T1 of Document object D1 is deemed to refer to Document objectD2 and Document object D3.

The term “related reference document subset” may refer to a data objectthat describes one or more related reference documents for acorresponding portion of an index document in accordance with aninferred annotative relationship for the corresponding portion. Forexample, if a text portion T1 of Document object D1 is deemed to referto Document object D2, the related reference document subset for T1includes D2. As another example, if a text portion T1 of Document objectD1 is deemed to refer to Document object D2 and Document object D3, therelated reference document subset for T1 includes D2 and D3.

The term “related reference document sequence” may refer to a dataobject that describes an inferred sequence of the set of relateddocument objects for a particular index document, where the inferredsequence defines an inferred ordering value for each related documentobject in the set of related document objects. For example, if adocument object D1 includes a text portion T1 that includes before atext portion T2, and if T1 is deemed to refer to document object D2while T2 is deemed to refer to document object D3, then the relatedreference document sequence for the document object D1 may describe thatD2 has a lower ordering value relative to D3.

The term “cross-document annotation machine learning model” may refer toa data object that describes parameters and/or hyper-parameters of amachine learning model that is trained using a set of prior annotativerelationships in order to, once trained, generate a set of inferredannotative relationships for an input document object. For example, across-document annotation machine learning model may be a supervisedmachine learning model that is configured to process a set of priorannotative relationships that include a set of manually-generated priorannotative relationships in order to generate a set of inferredannotative relationships that include a set of automatically-generatedprior prior annotative machine learning models. In some embodiments, byutilizing a cross-document annotation machine learning model to generatetraining data for an entity-document correlation machine learning model,various embodiments of the present invention enable semi-supervisedtraining of an entity-document correlation machine learning model.

The term “optimal document sequence” may refer to a data object thatdescribes an optimal ordering of a set of related document objectsassociated with an input document object, where the optimal orderingassigns an optimal ordering value to each related document object in theset of related document objects, and where the optimal ordering isdetermined by processing the set of related document objects using adocument sequence optimization machine learning model. For instance, theoptimal document sequence in particular embodiments may indicate anordering of a set of document objects with respect to their use by areviewer agent during performance of a processing task (e.g., apre-authorization task) associated with the predictive entity.

The term “sequence-related features” may refer to a data object thatdescribes one or more features for each document object in a set ofrelated document objects for a corresponding input document object thatmay be used in identifying the optimal document sequence for the inputdocument object. In some embodiments, sequence-related features describeinformation and/or characteristics of document objects found in the setof related document objects for an input document object. Examples ofsequence-related features include document identifiers, document-typeidentifiers, document creation date identifiers, document latestrevision date identifiers, and/or the like.

The term “document sequence optimization machine learning model” mayrefer to a data object that describes parameters and/or hyper-parametersof a machine learning model that processes a related set of documentobjects for an input document object to generate the optimal documentsequence for the input document object. For instance, in particularembodiments, the document sequence optimization machine learning modelmay be configured to generate a set of candidate document sequences fora set of input document objects, generate a combined sequenceprobability value for each candidate document sequence of the set ofcandidate document sequences, and select the candidate document sequencewith the highest combined sequence probability value as the optimaldocument sequence. In some embodiments, the term “candidate documentsequence” may refer to a candidate sequential ordering of the documentobjects found in the set of document objects. For example, given a setof related document objects that include a document object D1 and adocument object D2, the following candidate document sequences may beidentified: a candidate document sequence D1→D2 and a candidate documentsequence D2→D1.

The term “combined sequence probability value” may refer to a dataobject that describes an overall likelihood that a particular candidatedocument sequence is the optimal sequential ordering of the set ofrelated document objects for a corresponding input document object. Insome embodiments, given a candidate document sequence D₁→D₂→ . . .→D_(n), the combined sequence probability value for the candidatedocument sequence may be calculated by computing Π_(i=1)^(n)P(D_(i)|D_(i-1), D_(i-2), . . . , D₁).

The term “prior selection bonus” may refer to a data object thatdescribes a value that is configured to be added to the combinedsequence probability value for a particular candidate document sequence,where the value is determined based on frequency of past selection ofthe particular candidate document sequence as an optimal documentsequence. For example, in some embodiments, application of a priorselection bonus to the combined sequence probability value for aparticular candidate document sequence may be configured to reward arepeat selection of the particular candidate document sequence becausethe particular candidate document sequence is known to have beenselected for a past pre-authorization case, where the value isdetermined based on frequency of past selection of the particularcandidate document sequence as an optimal document sequence.

Overview

Many processes found throughout a number of different industries arerequired to gather and analyse relevant data from a number of differentdata sources. Oftentimes these sources may involve various entities bothinternal and external to an organization and may take various forms suchas files, repositories, databases, websites, systems, services, and thelike. For example, many health insurance providers use apre-authorization process with respect to customers (e.g., members) whoare seeking pre-authorization for a medical procedure and/or treatment.The pre-authorization process is typically carried out by an employee(e.g., a reviewer) who gathers and analyses information with respect tothe pre-authorization to determine whether to provide authorization forthe medical procedure and/or treatment. This information may pertainspecifically to the customer such as the insurance policy that customercurrently carries as well as demographics on the customer, or moregenerally to aspects of the pre-authorization process such as generalguidelines, state and federal mandates, general information on themedical procedure and/or treatment, and the like. This information mayneed to be gathered from several different sources as well as need to begathered in a particular sequential order since certain information thatis gathered can then be used to gather and/or analyse furtherinformation used in the process.

As a result, the pre-authorization process employed by many healthinsurance providers can serve as a bottleneck because of the timeinvolved in performing the process and many customers may have to wait aconsiderable amount of time to receive authorization for a particularmedical procedure and/or treatment. This wait can oftentimes beproblematic when the medical procedure and/or treatment may be urgent.Furthermore, the bottleneck can lead to many inefficiencies with respectto the health insurance provider's limited resources such as personnelwho are required to perform the process and/or systems that are requiredto gather the information from the different sources.

Processes in other industries may experience the same bottleneck due tohaving to obtain and analyse information from a number of differentsources. For example, an Internet service provider may have an extensivenetwork system used in providing Internet service to its customers. Insome embodiments, the network system may involve various servers,routers, and other components that are located over a large area. TheInternet service provider may have a troubleshooting process fortroubleshooting issues that occur within the network system.Accordingly, this process may involve gathering and analysinginformation from certain components within the network system when anissue occurs as well as information from various sources that providesdocumentation on addressing the issue. In addition, this information mayneed to be gathered and/or analysed in a particular sequential order.For example, information on Quality-of-Service may need to be initiallygathered and analysed to identify a particular component (e.g., arouter) within the system network having a problem. Information may thenneed to be gathered and analysed from the component to identify thespecific issue. From there, further information may need to be gatheredand analysed from various technical manuals to identify an approach toaddress the issue, all of which can take a considerable amount of timeresulting in a bottleneck in restoring Internet service to itscustomers.

This may still be the case even in instances in which thetroubleshooting process is being carried out by automation (e.g.,artificial intelligence) instead of a human. The troubleshooting systemis still required to gather and analyse the information from thedifferent sources in a sequential manner to resolve the issue. Althoughthe troubleshooting system may be able to identify and resolve the issuefaster than a human, the troubleshooting process carried out by thetroubleshooting system can still lead to a time delay in restoringInternet service to customers that is considered unacceptable.

Thus, embodiments of the present invention provide concepts forreplacing the existing processes of aggregating (e.g., gathering andanalyzing) information from different sources that is oftentimesmanually-driven with an improved automated process. In variousembodiments, a machine-learning model is used in identifying a set ofdocument objects representing different sources having information thatis needed in performing a certain process and/or task. In addition, anoptimal document sequence may be identified for the set of documentobjects that provides an optimized ordering of the document objectsfound in the set for use in performing the process and/or task.Accordingly, in particular embodiments, a machine-learning model may beused in identifying the optimal document sequence for the set ofdocuments. The information needed from each of the sources identified bythe document objects may then be retrieved and provided in aconsolidated form according to the optimal document sequence. Thisconsolidated form may then be used in performing the process and/ortask.

The disclosed solution provided herein is more effective, accurate, andfaster than aggregating information from different sources performed bya human and aggregating information from different sources performed byautomated processes. Further, the deterministic prioritization rules,probabilistic prioritization rules, and machine learning model(s) cancarry out complex mathematical operations that cannot be performed bythe human mind (e.g., determining a likelihood of documents being neededfor a specific process and/or task). Especially when such operationsneed to be performed in a short timeframe. In doing so, variousembodiments of the present disclosure make major technical contributionsto improving the efficiency and reliability of existing processes foraggerating information from different sources.

An exemplary overview is now provided to demonstrate various embodimentsof the disclosure provided herein for aggregating information fromvarious information sources to be used with respect to a predictiveentity. An example of such an entity is now introduced that is usedthroughout the remainder of the disclosure to assist the reader'sunderstanding of various embodiments of the disclosure. It should beunderstood that this example is provided for the purpose of facilitatingthe reader's understanding of embodiments and should not be interpretedto limit the scope of disclosure.

The example involves using various embodiments in aggregatinginformation from various information sources to be used in apre-authorization process performed by a health insurance provider foradjudicating cases in which members are seeking pre-authorization forvarious medical procedures and/or treatments. The purpose of the processis to ensure the medical procedures and/or treatments are properlycovered under the members' insurance policies and to what extent theyare covered. In some embodiments, a reviewer who is employed by thehealth insurance provider performs the pre-authorization process uponreceiving a request for a member who is seeking authorization for aparticular medical procedure and/or treatment (instance).

FIG. 1 an overview of a process 100 for aggregating information fromdifferent sources to be used for the incoming case is provided thatmakes use of various embodiments of the disclosure. Accordingly, therequest is treated as a new incoming case that can be received over amultitude of different channels 110 such as, for example, a web portal,a fax, a mobile application, manually, through a phone call, and thelike. Therefore, the case 115 is captured in the health insuranceprovider's system and the initial information provided along with thecase 115 may be checked to ensure the minimal amount of information ispresent to begin the pre-authorization process. For example, the minimumamount of information necessary to begin the process may includeidentification information for the member, CPT and diagnosis codes, andthe place of service for the medical procedure and/or treatment. Inaddition, case enrichment may occur that involves supplementing theinitial information from information from various sources 120 such asdatabases and/or repositories to ready the case for successfuladjudication.

At this point, a document prediction process 125 is performed to predicta subset of related document objects that represent sources havinginformation needed during the pre-authorization process so that thereviewer can successfully adjudicate the case. In some instances, thedocument prediction process 125 may also provide a related documentsequence providing an order in which the document objects found in thesubset are to be processed. In some embodiments, in particularembodiments, an entity-document correlation machine-learning model isused to identify the subset of related document objects in which variousinformation associated with the case may be used as input to the model.As discussed further herein, depending on the embodiment, thismachine-learning model may be configured simply to provide the subset ofrelated document objects or may be configured to provide the subsetidentified with the related document sequence.

Further, the document prediction process 125 in particular embodimentsmay include a document sequence optimization machine-learning model thatorders the set of document objects in an optimal document sequence. Insome embodiments, the ordering may be “optimized” so that the documentobjects are provided in a particular sequence with respect to wheninformation from the sources represented by the document objects isneeded during the pre-authorization process. That is to say, the optimaldocument sequence identifies an order for the set of document objects sothat information found in each of the sources identified by the documentobjects is available to the reviewer when needed to conduct a successfuladjudication of the case.

Once the set of document objects have been identified, the process 100continues in various embodiments with data scraping and downloading 130each of the information sources identified in the set of documentobjects as required. This step in the process may involve downloadingand processing different types of information sources 135 such as, forexample, pdf files, scanned images, scanned pdf files, word processingdocuments, repositories, databases, HTML source files, websites and thelike.

In some embodiments, the information sources may be divided into twocategories. A first category of information sources may involve sourceshaving generic information that is not necessarily specific to theparticular case. For example, generic information may involve state andfederal health mandates and/or standard operating procedures. Thesegeneric sources may be downloaded, parsed, and stored in some type ofrepository so that their retrieval is centralized and streamlined. Inaddition, these generic sources may be updated on a periodic basis(e.g., weekly) so that any changes to the information found in thesources may be updated in the repository.

The second category of information sources is member specific sources.These are considered information sources holding information that isspecific to the particular member involved in the case. For example,records providing information on past medical examines conducted on themember and/or records on previous insurance claims submitted by themember. Accordingly, these documents are downloaded in real-time invarious embodiments as the pre-authorization process is taking place.

Once the information sources have been obtained, information found inone or more of the sources may be parsed. In some embodiments, inparticular embodiments, one or more custom parsers 140 may be utilizedto perform this operation. For instance, an Optimal CharacterRecognition (OCR) based parser may be used to parse information found inimages. For example, downloaded pdf files may be converted into imagesand given as input to the OCR based parser. In addition, other parsersmay be utilized such as an XML based parser, HTML based parser, wordprocessing based parser, and the like in parsing information fromdifferent information sources. Accordingly, in particular embodiments,the result of the parsing process is a dictionary for each informationsource containing all the information downloaded and parsed from thesource.

The process 100 then continues in various embodiments with extractingthe needed information 145 from the dictionaries for the varioussources. In some embodiments, in particular embodiments, the informationfound in the dictionary for each source is passed through a NaturalLanguage Processing (NLP) pipeline to extract named entities and theircorresponding context found in each source. Thus, Named-EntityRecognition (NER) may be used in embodiments to identify, locate, andclassify named entities mentioned in the information into pre-definedcategories. In addition, corresponding sections of the information(e.g., paragraphs) may also be identified/mapped in the process.

At this point, the named entities found in each of the sources can bemapped to a knowledge map. In generally, a knowledge map is a diagramthat depicts relationships between pieces of knowledge (e.g., subjects).Typically, a knowledge map represents the pieces of knowledge connectedwith labeled links (e.g., arrows) in a hierarchical structure with therelationships between the pieces of knowledge articulated in linkingphrases. Therefore, as discussed in further detail herein, asource-specific knowledge tree may be constructed in particularembodiments for each source based on annotations provided by experts onpast information gathered from the source.

Accordingly, specific queries to extract relevant sections of theinformation found in the dictionary for each source can be constructedbased on an inference performed on the corresponding knowledge map forthe document to find the entities for which the relevant sections of theinformation found in the dictionary should be extracted. For instance,in the example, the correct relationships can be selected form theknowledge map for a particular source containing information on policyrelated to certain treatments based on the policy related and treatmentrelated details found in the map. These relationships are parsed to getthe list of entities for which relevant sections of the source are to beextracted.

Finally, the relevant sections of information for each of the source isextracted from the corresponding dictionary based on the entitiesidentified from the knowledge tree and combined to form an aggregatecontaining all the relevant information from all of the documents. Thus,returning the example, the relevant information may be combined into abenefits coverage case note 150 that can then be used by the reviewer inadjudicating the pre-authorization case.

As a result, all of the relevant information needed by the reviewer inperforming the pre-authorization process is now found in a singledocument (the case note 150) so that the reviewer is not required togather the needed information on his or her own. In addition, inparticular embodiments, the information provided in the case note 150 isthe sequential manner as identified earlier in the process 100.Therefore, the information can be found in the case note 150 in theorder needed by reviewer in adjudicating the pre-authorization case 115in a timely fashion.

Exemplary Technical Contributions

Various embodiments of the present invention provide techniquesconfigured to increase efficiency and reliability of document retrievalin document management systems. Large document management systems with alarge repository of documents incur substantial costs associated withend-user attempts to detect and browse related documents. For example,when performing a particular task, an end user may need to retrieve aset of documents related to the noted task. In doing so, the end usermay need to perform computationally expensive operations to search fortarget documents, browse candidate documents to determine whether theyprovide the relevant information, and download documents of interest.Various embodiments of the present invention enable reducing suchcomputational costs associated with retrieval of target documents byintelligently inferring annotative/subject-matter-based relationshipsacross documents using entity-document correlation machine learningmodels and/or by intelligently inferring sequential/presentation-relatedrelationships between documents using document sequence optimizingmachine learning models. In doing so, various embodiments of the presentinvention make important technical contribution to increasing efficiencyand reliability of document retrieval in document management systems andsubstantially improve the field of distributed document retrieval indistributed document server systems.

Exemplary use cases for various embodiments of the present invention aredescribed below. For example, many processes carried out in variousindustries involve the use of data that is gathered from differentsources. For example, in the health insurance industry, many medicalprocedures and/or treatments require a member to first seekpre-authorization from his or her health insurance provider beforehaving the procedure and/or receiving the treatment. This is done toensure the member has the correct insurance coverage for the procedureand/or treatment. The pre-authorization process generally involves anindividual (e.g., a reviewer) gathering information typically from aplethora of sources to review the case and determine whether or not toprovide authorization. For example, the reviewer may be required toreview guidelines, state and/or federal mandates, and/or CurrentProcedural Terminology (CPT) code-based standard operating proceduredocuments. Accordingly, this information may involve a number ofdifferent sources such as different repositories, websites, libraries,databases, etc.

Thus, a reviewer who is investigating a single prior-authorization casefor a health insurance provider may have to access and navigate severaldifferent sources manually to find needed information. This navigationprocess is time-consuming as the reviewer is oftentimes required totoggle between different sources. In addition, many of the sources havea considerable amount of information that the reviewer is then requiredto search through to find what he or she needs for the case, which addsmore time to the pre-authorization process. Further, the sourcestypically need be analyzed in a particular order (e.g., sequentially)because information extracted from one source may be needed to analyzeanother source, all of which adds even more time to thepre-authorization process. As a result, the pre-authorization processpracticed by many health insurance providers results in a hugebottleneck in providing their customers with timely service. Inaddition, the pre-authorization process results in many inefficiencieswith respect to use of resources such as personnel and processingsystems needed to carry out the process.

The same is true with respect to other processes carried out in otherindustries involving the gathering and analyzing of information fromvarious sources. For example, troubleshooting processes used in manytechnology industries for identifying problems in computer systemsand/or networks involve gathering and analyzing information from severaldifferent sources. For instance, a troubleshooting process used by anInternet service provider may involve gathering information from severaldifferent sources (e.g., servers, routers, and/or WiFi satellite) withinthe provider's network structure, as well as sources to aid introubleshooting the problem (Quality-of-Service logs and/or technicalmanuals). In some embodiments, a technician who is working for theInternet service provider or analytics system (e.g., ArtificialIntelligence) used by the provider may be required to access and analyzeinformation from all these different sources to identify and correct anetwork problem. Again, the information may need to be analyzed in aparticular order (e.g., sequentially). This can result in a hugebottleneck in correcting the problem in a timely manner. In addition,the troubleshooting process can result in many inefficiencies withrespect to the use of resources such as the technician and/or analyticssystem needed to carry out the process.

Therefore, a need exists in the industry for improved systems andmethods for aggregating information from various sources to aid instreamlining processes that make use of the information it is withrespect to these considerations and others that the disclosure herein ispresented.

Computer Program Products, Systems, Methods, and Computing Entities

Embodiments of the present invention may be implemented in various ways,including as computer program products that comprise articles ofmanufacture. Such computer program products may include one or moresoftware components including, for example, software objects, methods,data structures, and/or the like. A software component may be coded inany of a variety of programming languages. An illustrative programminglanguage may be a lower-level programming language such as an assemblylanguage associated with a particular hardware architecture and/oroperating system platform. A software component comprising assemblylanguage instructions may require conversion into executable machinecode by an assembler prior to execution by the hardware architectureand/or platform. Another example programming language may be ahigher-level programming language that may be portable across multiplearchitectures. A software component comprising higher-level programminglanguage instructions may require conversion to an intermediaterepresentation by an interpreter or a compiler prior to execution.

Other examples of programming languages include, but are not limited to,a macro language, a shell or command language, a job control language, ascript language, a database query or search language, and/or a reportwriting language. In one or more example embodiments, a softwarecomponent comprising instructions in one of the foregoing examples ofprogramming languages may be executed directly by an operating system orother software component without having to be first transformed intoanother form. A software component may be stored as a file or other datastorage construct. Software components of a similar type or functionallyrelated may be stored together such as, for example, in a particulardirectory, folder, or library. Software components may be static (e.g.,pre-established or fixed) or dynamic (e.g., created or modified at thetime of execution).

A computer program product may include a non-transitorycomputer-readable storage medium storing applications, programs, programmodules, scripts, source code, program code, object code, byte code,compiled code, interpreted code, machine code, executable instructions,and/or the like (also referred to herein as executable instructions,instructions for execution, computer program products, program code,and/or similar terms used herein interchangeably). Such non-transitorycomputer-readable storage media include all computer-readable media(including volatile and non-volatile media).

In one embodiment, a non-volatile computer-readable storage medium mayinclude a floppy disk, flexible disk, hard disk, solid-state storage(SSS) (e.g., a solid state drive (SSD), solid state card (SSC), solidstate module (SSM), enterprise flash drive, magnetic tape, or any othernon-transitory magnetic medium, and/or the like. A non-volatilecomputer-readable storage medium may also include a punch card, papertape, optical mark sheet (or any other physical medium with patterns ofholes or other optically recognizable indicia), compact disc read onlymemory (CD-ROM), compact disc-rewritable (CD-RW), digital versatile disc(DVD), Blu-ray disc (BD), any other non-transitory optical medium,and/or the like. Such a non-volatile computer-readable storage mediummay also include read-only memory (ROM), programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), flash memory (e.g.,Serial, NAND, NOR, and/or the like), multimedia memory cards (MMC),secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF)cards, Memory Sticks, and/or the like. Further, a non-volatilecomputer-readable storage medium may also include conductive-bridgingrandom access memory (CBRAM), phase-change random access memory (PRAM),ferroelectric random-access memory (FeRAM), non-volatile random-accessmemory (NVRAM), magnetoresistive random-access memory (MRAM), resistiverandom-access memory (RRAM), Silicon-Oxide-Nitride-Oxide-Silicon memory(SONOS), floating junction gate random access memory (FJG RAM),Millipede memory, racetrack memory, and/or the like.

In one embodiment, a volatile computer-readable storage medium mayinclude random access memory (RAM), dynamic random access memory (DRAM),static random access memory (SRAM), fast page mode dynamic random accessmemory (FPM DRAM), extended data-out dynamic random access memory (EDODRAM), synchronous dynamic random access memory (SDRAM), double datarate synchronous dynamic random access memory (DDR SDRAM), double datarate type two synchronous dynamic random access memory (DDR2 SDRAM),double data rate type three synchronous dynamic random access memory(DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), TwinTransistor RAM (TTRAM), Thyristor RAM (T-RAM), Zero-capacitor (Z-RAM),Rambus in-line memory module (RIMM), dual in-line memory module (DIMM),single in-line memory module (SIMM), video random access memory (VRAM),cache memory (including various levels), flash memory, register memory,and/or the like. It will be appreciated that where embodiments aredescribed to use a computer-readable storage medium, other types ofcomputer-readable storage media may be substituted for or used inaddition to the computer-readable storage media described above.

As should be appreciated, various embodiments of the present inventionmay also be implemented as methods, apparatus, systems, computingdevices, computing entities, and/or the like. As such, embodiments ofthe present invention may take the form of a data structure, apparatus,system, computing device, computing entity, and/or the like executinginstructions stored on a computer-readable storage medium to performcertain steps or operations. Thus, embodiments of the present inventionmay also take the form of an entirely hardware embodiment, an entirelycomputer program product embodiment, and/or an embodiment that comprisescombination of computer program products and hardware performing certainsteps or operations.

Embodiments of the present invention are described below with referenceto block diagrams and flowchart illustrations. Thus, it should beunderstood that each block of the block diagrams and flowchartillustrations may be implemented in the form of a computer programproduct, an entirely hardware embodiment, a combination of hardware andcomputer program products, and/or apparatus, systems, computing devices,computing entities, and/or the like carrying out instructions,operations, steps, and similar words used interchangeably (e.g., theexecutable instructions, instructions for execution, program code,and/or the like) on a computer-readable storage medium for execution.For example, retrieval, loading, and execution of code may be performedsequentially such that one instruction is retrieved, loaded, andexecuted at a time. In some exemplary embodiments, retrieval, loading,and/or execution may be performed in parallel such that multipleinstructions are retrieved, loaded, and/or executed together. Thus, suchembodiments can produce specifically configured machines performing thesteps or operations specified in the block diagrams and flowchartillustrations. Accordingly, the block diagrams and flowchartillustrations support various combinations of embodiments for performingthe specified instructions, operations, or steps.

Exemplary System Architecture

FIG. 2 provides an illustration of a system architecture 200 that may beused in accordance with various embodiments of the disclosure. In someembodiments, the architecture 200 includes various components involvedin aggregating information from various information sources that are tobe used in performing a task. Accordingly, the components may includeone or more application servers 210 that may be in communication withand one or more information sources 215, 220, 225 over one or morenetworks 230. It should be understood that the application server(s) 210may be made up of several servers, storage media, layers, and/or othercomponents, which may be chained or otherwise configured to interactand/or perform tasks. Specifically, the application server(s) 210 mayinclude any appropriate hardware and/or software for interacting withthe information sources 215, 220, 225 as needed to execute aspects ofone or more applications for conducting processes that involveaggregating information found in the sources 215, 220, 225 and handlingdata access and business logic for such.

In addition, the architecture 200 may include one or more computingdevices 235 used by individuals to conduct one or processes that makeuse of aggregated information. For example, the computing devices 235may be used by reviewer(s) for a health insurance provider in conductingan analysis on pre-authorization cases. In some embodiments, thedevice(s) 235 may be one of many different types of devices such as, forexample, a desktop or laptop computer or a mobile device such as a smartphone or tablet.

As noted, the application server(s) 210, information sources 215, 220,225, and computing device(s) 235 may communicate with one another overone or more networks 230. Depending on the embodiment, these networks230 may comprise any type of known network such as a land area network(LAN), wireless land area network (WLAN), wide area network (WAN),metropolitan area network (MAN), wireless communication network, theInternet, etc., or combination thereof. In addition, these networks 230may comprise any combination of standard communication technologies andprotocols. For example, communications may be carried over the networks230 by link technologies such as Ethernet, 802.11, CDMA, 3G, 4G, ordigital subscriber line (DSL). Further, the networks 230 may support aplurality of networking protocols, including the hypertext transferprotocol (HTTP), the transmission control protocol/internet protocol(TCP/IP), or the file transfer protocol (FTP), and the data transferredover the networks 230 may be encrypted using technologies such as, forexample, transport layer security (TLS), secure sockets layer (SSL), andinternet protocol security (IPsec). Those skilled in the art willrecognize FIG. 2 represents but one possible configuration of a systemarchitecture 200, and that variations are possible with respect to theprotocols, facilities, components, technologies, and equipment used.

Exemplary Computing Entity

FIG. 3 provides a schematic of a computing entity 300 that may be usedin accordance with various embodiments of the present invention. Forinstance, the computing entity 300 may be one or more of the applicationservers 210 and/or devices 235 found in the system architecture 200previously described in FIG. 2 . In general, the terms computing entity,entity, device, system, and/or similar words used herein interchangeablymay refer to, for example, one or more computers, computing entities,desktop computers, mobile phones, tablets, phablets, notebooks, laptops,distributed systems, items/devices, terminals, servers or servernetworks, blades, gateways, switches, processing devices, processingentities, set-top boxes, relays, routers, network access points, basestations, the like, and/or any combination of devices or entitiesadapted to perform the functions, operations, and/or processes describedherein. Such functions, operations, and/or processes may include, forexample, transmitting, receiving, operating on, processing, displaying,storing, determining, creating/generating, monitoring, evaluating,comparing, and/or similar terms used herein interchangeably. In oneembodiment, these functions, operations, and/or processes can beperformed on data, content, information, and/or similar terms usedherein interchangeably.

Although illustrated as a single computing entity, those of ordinaryskill in the art should appreciate that the computing entity 300 shownin FIG. 3 may be embodied as a plurality of computing entities, tools,and/or the like operating collectively to perform one or more processes,methods, and/or steps. As just one non-limiting example, the computingentity 300 may comprise a plurality of individual data tools, each ofwhich may perform specified tasks and/or processes.

Depending on the embodiment, the computing entity 300 may include one ormore network and/or communications interfaces 325 for communicating withvarious computing entities, such as by communicating data, content,information, and/or similar terms used herein interchangeably that canbe transmitted, received, operated on, processed, displayed, stored,and/or the like. Thus, in certain embodiments, the computing entity 300may be configured to receive data from one or more data sources and/ordevices as well as receive data indicative of input, for example, from adevice.

The networks used for communicating may include, but are not limited to,any one or a combination of different types of suitable communicationsnetworks such as, for example, cable networks, public networks (e.g.,the Internet), private networks (e.g., frame-relay networks), wirelessnetworks, cellular networks, telephone networks (e.g., a public switchedtelephone network), or any other suitable private and/or publicnetworks. Further, the networks may have any suitable communicationrange associated therewith and may include, for example, global networks(e.g., the Internet), MANs, WANs, LANs, or PANs. In addition, thenetworks may include any type of medium over which network traffic maybe carried including, but not limited to, coaxial cable, twisted-pairwire, optical fiber, a hybrid fiber coaxial (HFC) medium, microwaveterrestrial transceivers, radio frequency communication mediums,satellite communication mediums, or any combination thereof, as well asa variety of network devices and computing platforms provided by networkproviders or other entities.

Accordingly, such communication may be executed using a wired datatransmission protocol, such as fiber distributed data interface (FDDI),digital subscriber line (DSL), Ethernet, asynchronous transfer mode(ATM), frame relay, data over cable service interface specification(DOCSIS), or any other wired transmission protocol. Similarly, thecomputing entity 300 may be configured to communicate via wirelessexternal communication networks using any of a variety of protocols,such as general packet radio service (GPRS), Universal MobileTelecommunications System (UMTS), Code Division Multiple Access 2000(CDMA2000), CDMA2000 1× (1×RTT), Wideband Code Division Multiple Access(WCDMA), Global System for Mobile Communications (GSM), Enhanced Datarates for GSM Evolution (EDGE), Time Division-Synchronous Code DivisionMultiple Access (TD-SCDMA), Long Term Evolution (LTE), Evolved UniversalTerrestrial Radio Access Network (E-UTRAN), Evolution-Data Optimized(EVDO), High Speed Packet Access (HSPA), High-Speed Downlink PacketAccess (HSDPA), IEEE 802.11 (Wi-Fi), Wi-Fi Direct, 802.16 (WiMAX),ultra-wideband (UWB), infrared (IR) protocols, near field communication(NFC) protocols, Wibree, Bluetooth protocols, wireless universal serialbus (USB) protocols, and/or any other wireless protocol. The computingentity 300 may use such protocols and standards to communicate usingBorder Gateway Protocol (BGP), Dynamic Host Configuration Protocol(DHCP), Domain Name System (DNS), File Transfer Protocol (FTP),Hypertext Transfer Protocol (HTTP), HTTP over TLS/SSL/Secure, InternetMessage Access Protocol (IMAP), Network Time Protocol (NTP), Simple MailTransfer Protocol (SMTP), Telnet, Transport Layer Security (TLS), SecureSockets Layer (SSL), Internet Protocol (IP), Transmission ControlProtocol (TCP), User Datagram Protocol (UDP), Datagram CongestionControl Protocol (DCCP), Stream Control Transmission Protocol (SCTP),HyperText Markup Language (HTML), and/or the like.

In addition, in various embodiments, the computing entity 300 includesor is in communication with one or more processing elements 310 (alsoreferred to as processors, processing circuitry, and/or similar termsused herein interchangeably) that communicate with other elements withinthe computing entity 300 via a bus 330, for example, or networkconnection. As will be understood, the processing element 310 may beembodied in several different ways. For example, the processing element310 may be embodied as one or more complex programmable logic devices(CPLDs), microprocessors, multi-core processors, coprocessing entities,application-specific instruction-set processors (ASIPs), and/orcontrollers. Further, the processing element 310 may be embodied as oneor more other processing devices or circuitry. The term circuitry mayrefer to an entirely hardware embodiment or a combination of hardwareand computer program products. Thus, the processing element 310 may beembodied as integrated circuits, application specific integratedcircuits (ASICs), field programmable gate arrays (FPGAs), programmablelogic arrays (PLAs), hardware accelerators, other circuitry, and/or thelike. As will therefore be understood, the processing element 310 may beconfigured for a particular use or configured to execute instructionsstored in volatile or non-volatile media or otherwise accessible to theprocessing element 310. As such, whether configured by hardware,computer program products, or a combination thereof, the processingelement 310 may be capable of performing steps or operations accordingto embodiments of the present invention when configured accordingly.

In various embodiments, the computing entity 300 may include or be incommunication with non-volatile media (also referred to as non-volatilestorage, memory, memory storage, memory circuitry and/or similar termsused herein interchangeably). For instance, the non-volatile storage ormemory may include one or more non-volatile storage or memory media 320such as hard disks, ROM, PROM, EPROM, EEPROM, flash memory, MMCs, SDmemory cards, Memory Sticks, CBRAM, PRAM, FeRAM, RRAM, SONOS, racetrackmemory, and/or the like. As will be recognized, the non-volatile storageor memory media 320 may store files, databases, database instances,database management system entities, images, data, applications,programs, program modules, scripts, source code, object code, byte code,compiled code, interpreted code, machine code, executable instructions,and/or the like. The term database, database instance, databasemanagement system entity, and/or similar terms used hereininterchangeably and in a general sense to refer to a structured orunstructured collection of information/data that is stored in acomputer-readable storage medium.

In particular embodiments, the memory media 320 may also be embodied asa data storage device or devices, as a separate database server orservers, or as a combination of data storage devices and separatedatabase servers. Further, in some embodiments, the memory media 320 maybe embodied as a distributed repository such that some of the storedinformation/data is stored centrally in a location within the system andother information/data is stored in one or more remote locations.Alternatively, in some embodiments, the distributed repository may bedistributed over a plurality of remote storage locations only. Asalready discussed, various embodiments contemplated herein useinformation sources 215, 220, 225 in which some or all theinformation/data required for various embodiments of the invention maybe stored.

In various embodiments, the computing entity 300 may further include orbe in communication with volatile media (also referred to as volatilestorage, memory, memory storage, memory circuitry and/or similar termsused herein interchangeably). For instance, the volatile storage ormemory may also include one or more volatile storage or memory media 315as described above, such as RAM, DRAM, SRAM, FPM DRAM, EDO DRAM, SDRAM,DDR SDRAM, DDR2 SDRAM, DDR3 SDRAM, RDRAM, RIMM, DIMM, SIMM, VRAM, cachememory, register memory, and/or the like. As will be recognized, thevolatile storage or memory media 315 may be used to store at leastportions of the databases, database instances, database managementsystem entities, data, images, applications, programs, program modules,scripts, source code, object code, byte code, compiled code, interpretedcode, machine code, executable instructions, and/or the like beingexecuted by, for example, the processing element 310. Thus, thedatabases, database instances, database management system entities,data, images, applications, programs, program modules, scripts, sourcecode, object code, byte code, compiled code, interpreted code, machinecode, executable instructions, and/or the like may be used to controlcertain aspects of the operation of the computing entity 300 with theassistance of the processing element 310 and operating system.

As will be appreciated, one or more of the computing entity's componentsmay be located remotely from other computing entity components, such asin a distributed system. Furthermore, one or more of the components maybe aggregated and additional components performing functions describedherein may be included in the computing entity 300. Thus, the computingentity 300 can be adapted to accommodate a variety of needs andcircumstances.

Exemplary System Operations

The logical operations described herein may be implemented (1) as asequence of computer implemented acts or one or more program modulesrunning on a computing system and/or (2) as interconnected machine logiccircuits or circuit modules within the computing system. Theimplementation is a matter of choice dependent on the performance andother requirements of the computing system. Accordingly, the logicaloperations described herein are referred to variously as states,operations, structural devices, acts, or modules. These operations,structural devices, acts, and modules may be implemented in software, infirmware, in special purpose digital logic, and any combination thereof.Greater or fewer operations may be performed than shown in the figuresand described herein. These operations may also be performed in adifferent order than those described herein. For example, anentity-document correlation machine learning model may be used without adocument sequence optimization machine learning model, a documentsequence optimization machine learning model without an entity-documentcorrelation machine learning model, training data for training anentity-document correlation machine learning model may be generatedwithout using a cross-document annotation machine learning model, and across-document annotation machine learning model may be used to generatetraining data for machine learning models other than an entity-documentcorrelation machine learning model.

Predicting Document Sequences for Input Documents

As previously discussed, various embodiments of the invention mayinvolve the use of one or more machine-learning models in identifying(e.g., predicting) a subset of related document objects identifyinginformation sources needed in performing a task and in some instances,an optimal document sequence for the identified subset of relateddocument objects. For instance, returning the example involving thepre-authorization process used within an health insurance provider, oneor more machine-learning models may be used in identifying andsequencing a subset of related document objects needed by a reviewer inadjudicating a particular pre-authorization case for a medical procedureand/or treatment.

Turning now to FIG. 4 , an overview of a process 400 for identifying asequence of related document objects for an input document object isprovided that makes use of machine-learning models in accordance withvarious embodiments of the disclosure. Again, the overview is discussedwith respect to receiving an incoming case 115 involving a memberseeking pre-authorization for a medical procedure and/or treatment fromhis or her health insurance provider. Therefore, as previouslydiscussed, the pre-authorization case 115 may include initialinformation that is then supplemented with additional information fromone or more information sources to augment the features 410 of thepre-authorization case 115 into entity metadata features. These featuresmay then be read by a document prediction engine 415 that uses thefeatures as input to an entity-document correlation machine-learningmodel that predicts a subset of related document objects 420 from a setof reference document objects, where the subset of related documentobjects 420 are deemed to be related to the pre-authorization case 115and thus deemed needed by the reviewer in adjudicating thepre-authorization case 115.

In some embodiments, depending on the embodiment, the entity-documentcorrelation machine learning model may be any one of many differenttypes of machine-learning models. For instance, in particularembodiments, the entity-document correlation machine learning model maybe a multi-label classification model that is configured to provide aprobability with respect to each document object found in the set ofreference document objects as to its likelihood the document object isneeded by the reviewer during his or her review of the pre-authorizationcase 115. Therefore, any document object found to have a probabilityover a threshold value (e.g., over 0.05) may be identified by thedocument prediction engine 415 as being needed by the reviewer. It isnoted that in some embodiments the entity-document correlation machinelearning model may be a deep learning model such as a neural networkconfigured to not only predict the subset of related document objectsneeded by the reviewer in adjudicating the pre-authorization case 115,but also to predict a related document sequence for the subset ofdocument objects.

In some embodiments, training data used in training the entity-documentcorrelation machine learning model may be constructed in particularembodiments by manually annotating a corpus made up of previous indexdocument objects (e.g., previous case notes 150) to identify the setreference document objects that were referenced by reviewers inadjudicating previous pre-authorization cases, what sections ofinformation were used from the sources represented by the documentobjects, and/or the order (sequence) in which the information wasreferenced by the reviewers. For example, briefly turning to FIG. 5 , acase note 500 for a previous pre-authorization is shown that has beenannotated. In some embodiments, an annotation 510, 520, 530 has beenadded for each information source that was used by the reviewer inconducting the pre-authorization process to identify a related referencedocument subset. In addition, cross-document annotations 515, 525, 535have been added identifying the sections of information (text) used fromthe sources.

In some embodiments, the information sources that make up the relatedreference document subset for the case note 500 are listed in a sequencein which they were used during the pre-authorization process. Thus,training data can be generated from these annotations 510, 520, 530 suchas, for example, a related reference document sequence 540 for therelated reference document subset, as well as the cross-documentannotations 515, 525, 535 representing the information sections wereused from (e.g., map to) the different document objects 545.

In addition, various document entity metadata features associated withthe annotated case note 500 may be included along with the trainingdata. For example, such features may include member demographic detailssuch as name, member identifier, address, and the like, member policydetails such as set number, policy number, and the like, and memberdiagnosis details such as diagnosis codes, CPT codes, and the like.Accordingly, these features may represent the initial information and/oraugmented information gathered on the case when it was originallyreceived. Thus, these features can be used as inputs and the relatedreference document subset and/or the related reference document sequencecan be used as outputs in training the entity-document correlationmachine learning model to recognize one or more inferred annotativerelationships between the inputs and outputs.

In addition, a cross-document annotation machine learning model may bedeveloped and used to automatically annotate the previous index documentobjects in particular embodiments. For example, the cross-documentannotation machine learning model in some embodiments may be a randomforest based model. Such a model may be developed to enable asignificant size corpus can be developed for training theentity-document correlation machine learning model and to reduce thenumber of previous index document objects that need to be manuallyannotated. In some embodiments, manually-annotated (tagged) previousindex document objects describing prior annotative relationships may beprocessed through a natural language processing pipeline to createfeatures that are used to train the cross-document annotation machinelearning model. Once trained, the cross-document annotation machinelearning model can then be used to annotate untagged previous indexdocument objects.

Returning now to FIG. 4 , once the document prediction engine 415 hasidentified the subset of related document objects 420 needed by thereviewer, the process 400 continues in particular embodiments withidentifying an optimal document sequence 430 for the subset 420 using adocument sequence optimization machine learning model 425. Accordingly,the document sequence optimization machine learning model 425 may beconfigured based on a probabilistic model to predict the optimalsequence of the document objects found in the subset of related documentobjects 420 identifying the information sources required by the reviewerin adjudicating the incoming case 115. Therefore, the model 425 maygenerate a set of candidate document sequences for the subset of relateddocument objects 420 and produce a combined sequence probability valuefor each candidate.

Each candidate document sequence represents a permutation of a differentsequential arrangement of the document objects found in the subset andthe combined sequence probability value for a candidate documentsequence may provide a probability based on a likelihood that thecandidate document sequence is an optimal documents sequence. In someembodiments, the combined sequence probability value for a documentsequence D₁→ . . . →D_(n) may be determined by performing the operationsdepicted in Equation 1:

P(D ₁ . . . D _(n))=Π_(i=1) ^(n) P(D _(i) |D _(i-1) , D _(i-2) , . . . ,D ₁)=P(D ₁)=*P(D ₂ |D ₁)* . . . *P(D _(n-1) |D _(n-2) , . . . , D ₁)  Equation 1

Accordingly, in particular embodiments, generating the combined sequenceprobability value for a candidate document sequence may involvegenerating an individual occurrence probability for an initial inputdocument object of the subset of related document objects 420 that isassociated with a lower-most document ordering value of one or moredocument sequence ordering values defined by the candidate documentsequence. Accordingly, a conditional occurrence probability may then bedetermined for each document sequence ordering value for occurrence of apost-initial input document object of the set of input document objectsthat is associated with the current document sequence ordering valuefollowing each input document object of the subset of related documentobjects 420 having a lower document ordering value relative to thecurrent document ordering value.

In some embodiments, the combined sequence probability value for acandidate document sequence is determined based on the pair-wisedocument sequencing probability for each pair of document objects foundin the candidate document sequence. An ordered document pair may be madeup of one or more prior document objects and a posterior document objectin which pair-wise document sequencing probability describes anestimated probability that the one or more prior document objects in theordered document pair is immediately followed by the posterior documentobject in the ordered document pair. In various embodiments, theposterior document object represents one of the document objects foundin the candidate document sequence and the one or more prior documentobjects represent all of the document objects found prior to thatdocument object in the candidate document sequence. However, in someembodiments, a Markov assumption may be used to approximate the set ofreference document objects and therefore the one or more prior documentobjects may represent a smaller number of the document objects foundprior to the posterior document object. For instance, n-gram modelingmay be used in particular embodiments. In these particular instances,the plurality of pair-wise document sequencing probabilities may beestimated using maximum likelihood estimation.

Accordingly, the document sequence optimization model is configured invarious embodiments to select the candidate document sequence with thehighest probability value as the optimal document sequence 430 for thesubset of related document objects 420. At this point, the documentobjects found in the subset 420 are processed in the order found in theoptimal document sequence 430 to retrieve the information sourcesassociated with the objects and extract out the needed information fromeach source to construct the case note 150 for reviewer. In someembodiments, the case note 150 is constructed with the informationextracted from the sources in the same sequence as the optimal documentsequence 430 identified for the subset of related document objects 420.Such a construction enables the reviewer to reference the information ina fashion that allows the reviewer to adjudicate the pre-authorizationincoming case 115 in an efficient manner.

Aggregation Module

Turning now to FIG. 6 , additional details are provided regarding aprocess flow for aggregating information from different informationsources to produce a case note 150 according to various embodiments.FIG. 6 is a flow diagram showing an aggregation module for performingsuch functionality according to various embodiments of the disclosure.In some embodiments, the particular process flow 600 shown in FIG. 6 isconfigured to produce a case note 150 for use by a reviewer inadjudicating a pre-authorization case 115. However, those of ordinaryskill in the art should appreciate, in light of this disclosure, thatthe process flow 600 shown in FIG. 6 can be modified with respect toaggregating information for other applications.

Thus, the process flow 600 begins in various embodiments with theaggregation module receiving the incoming case 115 in Operation 610. Aspreviously noted, the incoming case 115 may be received via one anynumber of different channels such as through a web portal, sent as afax, entered through an application on a mobile device, through a directcall from a healthcare provider, and the like. Accordingly, the incomingcase 115 is entered into the health insurance provider's system and theaggregation module may be invoked to process the case 115.

Therefore, the aggregation module initially determines whether theincoming case 115 has the minimum required information to enableprocessing in Operation 615. For example, in particular embodiments, theaggregation module may determine whether the incoming case 115 at leasthas information identifying the member who is seeking pre-authorizationfor a medical procedure and/or a medical treatment, informationidentifying the medical procedure and/or treatment such as CPT and/ordiagnosis codes, and information on the facility providing the medicalprocedure and/or treatment. If not, then the aggregation module sets anerror for the incoming case 115 in Operation 620. For example, theaggregation module may place an error message in a log file indicatingthe incoming case 115 does not have the required minimum informationand/or the aggregation module may send an error message to personnel viaemail or some other source of communication.

However, if the incoming case 115 is determined to have the requiredminimum information, then the aggregation module generates a subset ofrelated document objects for the incoming case 115 in Operation 625. Aspreviously discussed, a related document object identifies aninformation source (e.g., a document object) having information neededby the reviewer in conducting the pre-authorization process for theincoming case. In particular embodiments, the aggregation moduleperforms this operation by invoking a document prediction module, suchas a document prediction module that uses a cross-document annotationmachine learning model. In turn, the document prediction module predictsthe subset of related document objects from a set of reference documentobjects using a machine learning model and returns the subset to theaggregation module. Depending on the embodiment, the document predictionmodule may return the subset along with a related document sequencedefining a reference document ordering value for each related referencedocument found in the subset.

At this point, the aggregation module in some embodiments identifies anoptimal document sequence in Operation 630. In some embodiments, theoptimal document sequence may indicate an ordering of the subset ofrelated document objects with respect to the corresponding informationsources use by the reviewer during the pre-authorization process. Theordering is “optimized” so that the document objects found in the subsetare provided in a particular sequence with respect to when theinformation sources identified by the objects are needed during thepre-authorization process. Accordingly, the aggregation module performsthis operation in particular embodiments by invoking a sequence moduleand as detailed further herein, the sequence module uses a machinelearning model to identify the optimal document sequence.

Next, the aggregation module selects a document object from the subsetof related document objects based on the optimal document sequence inOperation 635. The aggregation module then downloads and parses theinformation for the source identified by the selected document object inOperation 640. The aggregation module performs this operation in variousembodiments by invoking a parsing module. As discussed further herein,the parsing module downloads the information from the source identifiedby the selected document object and parses the information, if needed,to construct a dictionary for the document object having the informationfor the corresponding source.

Once the information has been downloaded (and parsed if needed), theaggregation module extracts the needed information for conducting thepre-authorization process from the dictionary in Operation 645. In someembodiments, in particular embodiments, the aggregation module performsthis particular operation by invoking an extraction module. As discussedfurther herein, the extraction module in various embodiments tags theinformation found in the dictionary for the selected document objectwith respect to different entities (e.g., topics) mentioned in theinformation and identifies what sections of the information need to beextracted from the dictionary by identifying the appropriate entitiesbased on a knowledge map. The aggregation module then constructs thecase note 150 for the incoming case 115 using the extracted informationfor the selected document object in Operation 650.

At this point, the aggregation module determines whether anotherdocument object is found in the subset of related document objects inOperation 655. If so, then the aggregation module returns to Operation635 and selects the next document object from the subset based on theoptimal document sequence and performs the operations just described toextract the information needed for the case note 150 for the newlyselected document object. Once all of the document objects found in thesubset of related document objects have been processed, the aggregationmodule saves the case note 150 in Operation 660.

The reviewer may now use the case note 150 generated by the aggregationmodule for the pre-authorization process. Accordingly, since theaggregation module selected the document objects from the subset ofrelated document objects in an order based on the optimal documentsequence, the module has constructed the case note 150 in a sequentialmanner with the extracted information for each of the document objectsin the order needed by the reviewer in adjudicating the case 115. Thedifferent modules invoked by embodiments of the aggregation module arenow discussed.

Document Prediction Module

Turning now to FIG. 7 , additional details are provided regardingpredicting a subset of related document objects needed for a newincoming case 115 according to various embodiments. FIG. 7 is a flowdiagram showing a document prediction module for performing suchfunctionality according to various embodiments of the disclosure. Aspreviously mentioned, the document prediction module may be invoked byanother module in particular embodiments to predict the subset ofrelated document objects such as, for example, the aggregation modulepreviously described. However, with that said, the document predictionmodule may not necessarily be invoked by another module and may executeas a stand-alone module in other embodiments.

The process flow 700 begins with the document prediction moduleenriching the initial information received for the incoming case 115 bypulling additional information for the case 115 from differentinformation sources in Operation 710. Accordingly, the documentprediction module may be configured in particular embodiments to pull ininformation from various information sources to supplement the case 115.For example, the document prediction module may retrieve clinicalinformation with respect to the medical procedure and/or treatment beingsought by the member that may be useful in identifying (predicting) whatinformation may be needed by the reviewer during the pre-authorizationprocess.

The document prediction module then predicts the subset of relateddocument objects for the case 115 in Operation 715. In some embodiments,in particular embodiments, the document prediction module providesentity metadata features found in the enriched information for the case115 as input to the entity-document correlation machine learning modeland in turn, the model provides a probability for each document objectfound in a set of reference document objects with respect to alikelihood information provided by the information source associatedwith the object is needed by the reviewer in adjudicating the particularincoming case 115. In some embodiments, the entity metadata features arecategorized into three main categories: demographic features;policy-related feature; and diagnosis features. The demographic featuresmay include information that identifies the specific member such as, forexample, the member's name, as well as information describing the membersuch as the member's sex, age, and/or region of the country the memberresides. The policy-specific features may include information such asset number, policy number, and the like. The diagnosis features mayinclude information such as CPT codes, diagnosis codes, and the like.

As previously discussed, depending on the embodiment, theentity-document correlation machine learning model may be configured tosimply provide the subset of related document objects and/or the modelmay be configured to provide a related document sequence for the subsetof related document objects. The related document sequence defines arelated document ordering value for each related document object in thesubset of related document objects. As previously discussed, the subsetof related document objects can then be used in extracting informationfrom the various sources identified by the document objects found in thesubset to construct a case note 150 for the case 115. In addition, therelated document sequence can be used in particular instances toconstruct the case note 150 to provide the information in an ordercorresponding to the related document sequence.

Sequence Module

Turning now to FIG. 8 , additional details are provided regardingpredicting an optimal document sequence for a subset of related documentobjects needed for a new incoming case 115 according to variousembodiments. FIG. 8 is a flow diagram showing a sequence module forperforming such functionality according to various embodiments of thedisclosure. As previously mentioned, the sequence module may be invokedby another module in particular embodiments to predict the optimaldocument sequence such as, for example, the aggregation modulepreviously described. However, with that said, the sequence module maynot necessarily be invoked by another module and may execute as astand-alone module in other embodiments.

The process flow 800 begins with the sequence module generating a set ofcandidate document sequences for the subset of related document objectsin Operation 810. Accordingly, a candidate document sequence mayrepresent a candidate sequential ordering of the document objects foundin the subset. For example, the subset of related document objects mayinclude document objects A, B, and C. In this example, the sequencemodule may generate the candidate document sequences as: (1) A, B, C;(2) A, C, B; (3) B, A, C; (4) B, C, A; (5) C, A, B; and (6) C, B, A.

In particular embodiments, the sequence module may be configured togenerate the set of candidate document sequences by scaling downpotential candidate document sequences using some type of criteria. Forexample, information from an information source having state mandates oncancer treatments may always be needed first by a reviewer who ishandling a pre-authorization process in which the member is seekingpre-authorization for chemotherapy treatment. Therefore, in thisexample, the sequence module may be configured to only generatecandidate document sequences with the document object for theinformation source in the first position. While in another example, thesequence module may be configured to use the related document sequencegenerated by the document prediction module in identifying candidatedocument sequences. Those of ordinary skill in the art can envisionother criteria that may be used in identifying candidate documentsequences in light of this disclosure.

Once the set of candidate document sequences has been generated, thesequence module selects one of the candidates in Operation 815 andgenerates a combined sequence probability value for the candidate usingthe document sequence optimization machine learning model previouslydiscussed in Operation 820. The combined sequence probability valuerepresents a probability with respect to the likelihood the selectedcandidate document sequence is the optimal sequential ordering of thedocument objects for use by the reviewer in adjudicating the incomingcase 115.

Accordingly, a set of sequence-related features is provided as input tothe document sequence optimization machine learning model that includesthe candidate to generate the combined sequence probability value forthe candidate. In addition, these features may provide informationand/or characteristics on individual document objects found in thesubset of related document objects. For example, the features mayinclude a document identifier for each document object (informationsource) found in the candidate, a document type identifier thatidentifies a particular document object (information source) as beingpart of a particular type of document objects such as, for example,various forms for healthcare providers (office visit summaries formembers, evaluations based on physicals performed on members, and/ordiagnosis notes produced for members), and/or other features for eachdocument object such as a document creation date identifier, a documentlatest revision date identifier, and the like.

In addition, in particular embodiments, the document sequenceoptimization machine learning model may be configured to multiply theraw combined sequence probability value with a prior selection bonus togenerate the combined sequence probability value for the candidate. Insome embodiments, the prior selection bonus may be used to weight thecombined sequence probability value for the candidate document sequencebased on the candidate being associated with a prior document sequencethat was used for one or more past similar incoming cases.

At this point, the segmentation module determines whether there isanother candidate document sequence in Operation 825. If so, then thesegmentation module returns to Operation 815, selects the next candidatedocument sequence, and generates the combined sequence probability valuefor the newly selected candidate. Once the segmentation module hasgenerated the combined sequence probability value for each candidatedocument sequence, the module selects the candidate with the highestcombined sequence probability value as the optimal candidate sequence inOperation 830.

Parsing Module

Turning now to FIG. 9 , additional details are provided for parsinginformation downloaded from an information source according to variousembodiments. FIG. 9 is a flow diagram showing a parsing module forperforming such functionality according to various embodiments of thedisclosure. As previously mentioned, the parsing module may be invokedby another module in particular embodiments to download and parse theinformation from a source such as, for example, the aggregation modulepreviously described. However, with that said, the parsing module maynot necessarily be invoked by another module and may execute as astand-alone module in other embodiments.

The process flow 900 begins with the parsing module downloading theinformation for source corresponding to the document object in Operation910. As previously discussed, information sources may be divided intotwo different categories. The first category may involve generic sourcesthat may contain information that is not necessarily specific to themember. For example, a generic information source may involve ahealthcare mandate or guidelines published by a particular state. Insome embodiments, these sources may be downloaded and parsedperiodically (e.g., once every week) to capture any changes made to theinformation provided by the source, and stored as a dictionary in anindexed repository. Therefore, if the document object is for such aninformation source, then the parsing module simply retrieves thedictionary for the source from the repository. The second category ofinformation sources involve sources that may have information that isspecific to the member. For example, a doctor's summary of a visit themember made to the doctor. Therefore, the information for these sourcesin various embodiments is generally downloaded in real-time during thepre-authorization process.

Once the information for the source has been downloaded, the parsingmodule determines whether the information needs to be parsed inOperation 915. For example, the parsing module may have downloadedmember-specific information in a pdf format from the source and theinformation found in the pdf may need to be extracted from the pdf.Therefore, if the downloaded information does need to be parsed, theparsing module parses the information in Operation 920.

For instance, in particular embodiments, the parsing module may make useof one more parsers configured to parse specific types of documents. Forexample, the parsing module may make use of an Optical CharterRecognition (OCR) based parser configured to parse pdf files convertedto images. In addition, the parsing module may make use of one or moreof an XML based parser, HTML based parser, word processing documentbased parser, and the like.

The parsing module then builds a dictionary from the information for thedocument in Operation 925. In some embodiments, in particularembodiments, the parsing module may place indexes within the dictionaryin a standard layout to facilitate location of specific informationwithin the dictionary when needed. For example, the dictionary for aninformation source having text may be indexed according to <pageno><headings><subheadings> followed by the information content. Whilethe dictionary for an information source involving a table may beindexed according to <page no><headings><subheadings><table><rowid><column id> with the information content. Accordingly, the dictionaryproduced for the information source may then be used in constructing thecase note 150 for the incoming case 115.

Extraction Module

Turning now to FIG. 10 , additional details are provided for extractingthe information needed for a document object according to variousembodiments. FIG. 10 is a flow diagram showing an extraction module forperforming such functionality according to various embodiments of thedisclosure. As previously mentioned, the extraction module may beinvoked by another module in particular embodiments to extract theinformation needed for a document object such as, for example, theaggregation module previously described. However, with that said, theextraction module may not necessarily be invoked by another module andmay execute as a stand-alone module in other embodiments.

The process flow 1000 begins with the extraction module processing thedictionary for the document object being through a natural languageprocessing pipeline to extract the named entities found in thedictionary and their corresponding text in Operation 1010. Thus, theextraction module is configured in particular embodiments to performNamed-Entity Recognition (NER) on the dictionary to identify, locate,and classify named entities mentioned in the information found in thedictionary into pre-defined categories. In addition, the extractionmodule may identify (map) the corresponding information for each entity.

The extraction module then maps the named entities found in thedictionary to a knowledge map in Operation 1015. In various embodiments,a knowledge map is constructed for each of the document objects found inthe set of reference documents. In some embodiments, a knowledge map fora document object captures all the different entities (e.g., topics) ofinterest for the corresponding information source. For instance, aknowledge map for a document object may be an ontology representationand/or a set of rules with respect to the different entities of interestfor the corresponding source. Accordingly, these maps may be constructedwith the help of experts who are familiar with the corresponding sourceor type of source along with the previous index document objects thathave been annotated.

Turning briefly to FIG. 11 , an example knowledge map 1100 is providedfor a document object. This particular map 1100 is for document Doc 11110 and includes several relationships 1115, 1120, 1125, 1130, 1135with respect to entities of interest. Each relationship 1115, 1120,1125, 1130, 1135 is in the form of a triplet whose format is defined as<subject> <relationship> <predicate>. For example, the map 1100 includesa relationship 1115 as the triplet <Doc D1> 1110<hasName> 1115<Doc Name>1140. Accordingly, these relationships 1115, 1120, 1125, 1130, 1135 canbe defined with the help of the experts and any two given subjects(e.g., entities of interest) can be combined with a relationship 1115,1120, 1125, 1130, 1135 to form a triplet.

Therefore returning to FIG. 10 , the extraction module can perform aninferencing on the knowledge map to determine the entities for whichrelevant information should be extracted from the dictionary for thedocument object. For example, based on policy related to the incomingcase 115 and related details on the medical procedure and/or treatmentto be received by the member (e.g., CPT codes and/or diagnosis codes),the extraction module may pick the corresponding relationships from theknowledge map. Accordingly, the extraction module parses theserelationships to identify the entities of interest.

At this point, in particular embodiments, the extraction moduleconstructs a query based on the identified entities of interest from theknowledge map in Operation 1020. The extraction module then extracts therelevant information from the dictionary for document object based onthe query in Operation 1025. As previously discussed, the extractedinformation is then used in constructing the case note 150 for theparticular incoming case 115.

CONCLUSION

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation.

1. A computer-implemented method for generating an optimal documentsequence for a set of input document objects, the computer-implementedmethod comprising: processing sets of sequence-related features for theset of input document objects using one or more processors and adocument sequence optimization machine learning model to generate theoptimal document sequence, wherein the document sequence optimizationmachine learning model is configured to: (i) generate a combinedsequence probability value for each candidate document sequence of a setof candidate document sequences for the set of input document objectsand (ii) select the optimal document sequence from the set of candidatedocument sequences, wherein generating the combined sequence probabilityvalue for a particular candidate document sequence comprises:generating, using the one or more processors, an individual occurrenceprobability for an initial input document object of the set of inputdocuments that is associated with a lower-most document ordering valueof one or more document sequence ordering values defined by thecandidate document sequence; for each current document ordering value ofthe one or more document sequence ordering values other than thelower-most document ordering value, determining, using the one or moreprocessors, a conditional occurrence probability for occurrence of apost-initial input document object of the set of input document objectsthat is associated with the current document ordering value followingeach input document object of the set of input document objects having alower document ordering value relative to the current document orderingvalue; and generating, using the one or more processors, the combinedsequence probability value based at least in part on the individualoccurrence probability for the initial input document object and eachconditional occurrence probability for a post-initial input documentobject of the set of input document objects; and performing, using theone or more processors, one or more prediction-based actions based atleast in part on the optimal document sequence.
 2. Thecomputer-implemented method of claim 1, wherein the sets of sequencerelated features comprise a set of sequence related features for eachinput document object in the set of input document objects.
 3. Thecomputer-implemented method of claim 1, wherein the document sequenceoptimization machine learning model is configured to generate the set ofcandidate document sequences.
 4. The computer-implemented method ofclaim 1, wherein the combined sequence probability value for a candidatedocument sequence represents the likelihood that the candidate documentsequence is the optimal document sequence.
 5. The computer-implementedmethod of claim 1, wherein selecting the optimal document sequence fromthe set of candidate document sequences comprises selecting thecandidate document sequence having a highest combined sequenceprobability value.
 6. The computer-implemented method of claim 1,wherein the set of related sequence-related features for an inputdocument object of the set of input documents comprises one or more of:(i) a document identifier of the input document object or (ii) adocument group identifier of the input document object.
 7. Thecomputer-implemented method of claim 1, wherein the set of relatedsequence-related features for an input document object of the set ofinput documents comprises one or more of: (i) a document type identifierof the input document object, (ii) a document creation date identifierof the input document object, or (iii) a document latest revision dateidentifier of the input document object.
 8. An apparatus for generatingan optimal document sequence for a set of input document objects, theapparatus comprising at least one processor and at least onenon-transitory memory including program code, the at least onenon-transitory memory and the program code configured to, with theprocessor, cause the apparatus to at least: process sets ofsequence-related features for the set of input document objects using adocument sequence optimization machine learning model to generate theoptimal document sequence, wherein the document sequence optimizationmachine learning model is configured to: (i) generate a combinedsequence probability value for each candidate document sequence of a setof candidate document sequences for the set of input document objects,and (ii) select the optimal document sequence from the set of candidatedocument sequences, wherein generating the combined sequence probabilityvalue for a particular candidate document sequence comprises: generatingan individual occurrence probability for an initial input documentobject of the set of input documents that is associated with alower-most document ordering value of one or more document sequenceordering values defined by the candidate document sequence; for eachcurrent document ordering value of the one or more document sequenceordering values other than the lower-most document ordering value,determining a conditional occurrence probability for occurrence of apost-initial input document object of the set of input document objectsthat is associated with the current document ordering value followingeach input document object of the set of input document objects having alower document ordering value relative to the current document orderingvalue; and generating the combined sequence probability value based atleast in part on the individual occurrence probability for the initialinput document object and each conditional occurrence probability for apost-initial input document object of the set of input document objects;and perform one or more prediction-based actions based at least in parton the optimal document sequence.
 9. The apparatus of claim 8, whereinthe sets of sequence related features comprise a set of sequence relatedfeatures for each input document object in the set of input documentobjects.
 10. The apparatus of claim 8, wherein the document sequenceoptimization machine learning model is configured to generate the set ofcandidate document sequences.
 11. The apparatus of claim 8, wherein thecombined sequence probability value for a candidate document sequencerepresents the likelihood that the candidate document sequence is theoptimal document sequence.
 12. The apparatus of claim 8, whereinselecting the optimal document sequence from the set of candidatedocument sequences comprises selecting the candidate document sequencehaving a highest combined sequence probability value.
 13. The apparatusof claim 8, wherein the set of related sequence-related features for aninput document object of the set of input documents comprises one ormore of: (i) a document identifier of the input document object or (ii)a document group identifier of the input document object.
 14. Theapparatus of claim 8, wherein the set of related sequence-relatedfeatures for an input document object of the set of input documentscomprises one or more of: (i) a document type identifier of the inputdocument object, (ii) a document creation date identifier of the inputdocument object, or (iii) a document latest revision date identifier ofthe input document object.
 15. A non-transitory computer storage mediumcomprising instructions for generating an optimal document sequence fora set of input document objects, the instructions being configured tocause one or more processors to at least perform operations configuredto: process sets of sequence-related features for the set of inputdocument objects using a document sequence optimization machine learningmodel to generate the optimal document sequence, wherein the documentsequence optimization machine learning model is configured to: (i)generate a combined sequence probability value for each candidatedocument sequence of a set of candidate document sequences for the setof input document objects, and (ii) select the optimal document sequencefrom the set of candidate document sequences, wherein generating thecombined sequence probability value for a particular candidate documentsequence comprises: generating an individual occurrence probability foran initial input document object of the set of input documents that isassociated with a lower-most document ordering value of one or moredocument sequence ordering values defined by the candidate documentsequence; for each current document ordering value of the one or moredocument sequence ordering values other than the lower-most documentordering value, determining a conditional occurrence probability foroccurrence of a post-initial input document object of the set of inputdocument objects that is associated with the current document orderingvalue following each input document object of the set of input documentobjects having a lower document ordering value relative to the currentdocument ordering value; and generating the combined sequenceprobability value based at least in part on the individual occurrenceprobability for the initial input document object and each conditionaloccurrence probability for a post-initial input document object of theset of input document objects; and perform one or more prediction-basedactions based at least in part on the optimal document sequence.
 16. Thenon-transitory computer storage medium of claim 15, wherein the sets ofsequence related features comprise a set of sequence related featuresfor each input document object in the set of input document objects. 17.The non-transitory computer storage medium of claim 15, wherein thedocument sequence optimization machine learning model is configured togenerate the set of candidate document sequences.
 18. The non-transitorycomputer storage medium of claim 15, wherein the combined sequenceprobability value for a candidate document sequence represents thelikelihood that the candidate document sequence is the optimal documentsequence.
 19. The non-transitory computer storage medium of claim 15,wherein selecting the optimal document sequence from the set ofcandidate document sequences comprises selecting the candidate documentsequence having a highest combined sequence probability value.
 20. Thenon-transitory computer storage medium of claim 15, wherein the set ofrelated sequence-related features for an input document object of theset of input documents comprises one or more of: (i) a documentidentifier of the input document object or (ii) a document groupidentifier of the input document object.